Masking our emotions: Emotion recognition and perceived intensity differ by race and use of medical masks.

Although medical masks have played a key role in decreasing the transmission of communicable disease, they simultaneously reduce the availability of nonverbal cues fundamental to social interaction. In the present study, we determined the collective impact of medical masks on emotional expression recognition and perceived intensity as a function of actor race. Participants completed an emotional expression recognition task involving stimuli with or without medical masks. Across six basic emotional facial expressions, medical masks were associated with significantly more emotional expression recognition errors. Overall, the effects associated with race varied depending on the emotion and appearance of masks. Whereas recognition accuracy was higher for White relative to Black actors for anger and sadness, the opposite pattern was observed for disgust. Medical mask-wearing exacerbated actor-race related recognition differences for anger and surprise, but attenuated these differences for fear. Emotional expression intensity ratings were significantly reduced for all emotions except fear, where masks were associated with increased perceived intensity. Masks further increased already higher intensity ratings for anger in Black versus White actors. In contrast, masks eliminated the tendency to give higher intensity ratings for Black versus White sad and happy facial expressions. Overall, our results suggest that the interaction between actor race and mask wearing status with respect to emotional expression judgements is complex, varying by emotion in both direction and degree. We consider the implications of these results particularly in the context of emotionally charged social contexts, such as in conflict, healthcare, and policing.

Robust holistic face processing in early childhood during the COVID-19 pandemic.

The timing of the developmental emergence of holistic face processing and its sensitivity to experience in early childhood are somewhat controversial topics. To investigate holistic face perception in early childhood, we used an online testing platform and administered a two-alternative forced-choice task to 4-, 5-, and 6-year-old children. The children saw pairs of composite faces and needed to decide whether the faces were the same or different. To determine whether experience with masked faces may have negatively affected holistic processing, we also administered a parental questionnaire to assess the children's exposure to masked faces during the COVID-19 pandemic. We found that all three age groups performed holistic face processing when the faces were upright (Experiment 1) but not when the faces were inverted (Experiment 2), that response accuracy increased with age, and that response accuracy was not related to degree of exposure to masked faces. These results indicate that holistic face processing is relatively robust in early childhood and that short-term exposure to partially visible faces does not negatively affect young children's holistic face perception.

Joint Masked Face Recognition and Temperature Measurement System Using Convolutional Neural Networks.

With the outbreak of COVID-19, epidemic prevention has become a way to prevent the spread of epidemics. Many public places, such as hospitals, schools, and office places, require disinfection and temperature measurement. To implement epidemic prevention systems and reduce the risk of infection, it is a recent trend to measure body temperature through non-contact sensing systems with thermal imaging cameras. Compared to fingerprints and irises, face recognition is accurate and does not require close contact, which significantly reduces the risk of infection. However, masks block most facial features, resulting in the low accuracy of face recognition systems. This work combines masked face recognition with a thermal imaging camera for use as an automated attendance system. It can record body temperature and recognize the person at the same time. Through the designed UI system, we can search the attendance information of each person. We not only provide the design method based on convolutional neural networks (CNNs), but also provide the complete embedded system as a real demonstration and achieve a 94.1% accuracy rate of masked face recognition in the real world. With the face recognition system combined with a thermal imaging camera, the purpose of screening body temperature when checking in at work can be achieved.

Face masks are less effective than sunglasses in masking face identity.

The effect of covering faces on face identification is recently garnering interest amid the COVID-19 pandemic. Here, we investigated how face identification performance was affected by two types of face disguise: sunglasses and face masks. Observers studied a series of faces; then judged whether a series of test faces, comprising studied and novel faces, had been studied before or not. Face stimuli were presented either without coverings (full faces), wearing sunglasses covering the upper region (eyes, eyebrows), or wearing surgical masks covering the lower region (nose, mouth, chin). We found that sunglasses led to larger reductions in sensitivity (d') to face identity than face masks did, while both disguises increased the tendency to report faces as studied before, a bias that was absent for full faces. In addition, faces disguised during either study or test only (i.e. study disguised faces, test with full faces; and vice versa) led to further reductions in sensitivity from both studying and testing with disguised faces, suggesting that congruence between study and test is crucial for memory retrieval. These findings implied that the upper region of the face, including the eye-region features, is more diagnostic for holistic face-identity processing than the lower face region.

Mask-wearing selectivity alters observers' face perception.

Face masks became prevalent across the globe as an efficient tool to stop the spread of COVID-19. A host of studies already demonstrated that masks lead to changes in facial identification and emotional expression processing. These changes were documented across ages and were consistent even with the increased exposure to masked faces. Notably, mask-wearing also changes the state of the observers in regard to their own bodies and other agents. Previous research has already demonstrated a plausible association between observers' states and their perceptual behaviors. Thus, an outstanding question is whether mask-wearing would alter face recognition abilities. To address this question, we conducted a set of experiments in which participants were asked to recognize non-masked faces (Experiment 1), masked faces (Experiment 2) and novel objects (Experiment 3) while they were either masked or unmasked. Mask wearing hindered face perception abilities but did not modulate object recognition ability. Finally, we demonstrated that the decrement in face perception ability relied on wearing the mask on distinctive facial features (Experiment 4). Together, these findings reveal a novel effect of mask-wearing on face recognition. We discuss these results considering the plausible effect of somatosensory stimulation on visual processing as well as the effect of involuntary perspective taking.

Investigating the impact of disposable surgical face-masks on face identity and emotion recognition in adults with autism spectrum disorder.

With the outburst of the COVID-19 pandemic, disposable surgical face-masks (DSFMs) have been widely adopted as a preventive measure. DSFMs hide the bottom half of the face, thus making identity and emotion recognition very challenging, both in typical and atypical populations. Individuals with autism spectrum disorder (ASD) are often characterized by face processing deficits; thus, DSFMs could pose even a greater challenge for this population compared to typically development (TD) individuals. In this study, 48 ASDs of level 1 and 110 TDs underwent two tasks: (i) the Old-new face memory task, which assesses whether DSFMs affect face learning and recognition, and (ii) the Facial affect task, which explores DSFMs' effect on emotion recognition. Results from the former show that, when faces were learned without DSFMs, identity recognition of masked faces decreased for both ASDs and TDs. In contrast, when faces were first learned with DSFMs, TDs but not ASDs benefited from a "context congruence" effect, that is, faces wearing DSFMs were better recognized if learned wearing DSFMs. In addition, results from the Facial affect task show that DSFMs negatively impacted specific emotion recognition in both TDs and ASDs, although differentially between the two groups. DSFMs negatively affected disgust, happiness and sadness recognition in TDs; in contrast, ASDs performance decreased for every emotion except anger. Overall, our study demonstrates a general, although different, disruptive effect on identity and emotion recognition both in ASD and TD population.

Face processing in the infant brain after pandemic lockdown.

The role of visual experience in the development of face processing has long been debated. We present a new angle on this question through a serendipitous study that cannot easily be repeated. Infants viewed short blocks of faces during fMRI in a repetition suppression task. The same identity was presented multiple times in half of the blocks (repeat condition) and different identities were presented once each in the other half (novel condition). In adults, the fusiform face area (FFA) tends to show greater neural activity for novel versus repeat blocks in such designs, suggesting that it can distinguish same versus different face identities. As part of an ongoing study, we collected data before the COVID-19 pandemic and after an initial local lockdown was lifted. The resulting sample of 12 infants (9-24 months) divided equally into pre- and post-lockdown groups with matching ages and data quantity/quality. The groups had strikingly different FFA responses: pre-lockdown infants showed repetition suppression (novel > repeat), whereas post-lockdown infants showed the opposite (repeat > novel), often referred to as repetition enhancement. These findings provide speculative evidence that altered visual experience during the lockdown, or other correlated environmental changes, may have affected face processing in the infant brain.

COVID-19 masks increase the influence of face recognition algorithm decisions on human decisions in unfamiliar face matching.

Face masks, recently adopted to reduce the spread of COVID-19, have had the unintended consequence of increasing the difficulty of face recognition. In security applications, face recognition algorithms are used to identify individuals and present results for human review. This combination of human and algorithm capabilities, known as human-algorithm teaming, is intended to improve total system performance. However, prior work has shown that human judgments of face pair similarity-confidence can be biased by an algorithm's decision even in the case of an error by that algorithm. This can reduce team effectiveness, particularly for difficult face pairs. We conducted two studies to examine whether face masks, now routinely present in security applications, impact the degree to which this cognitive bias is experienced by humans. We first compared the influence of algorithm's decisions on human similarity-confidence ratings in the presence and absence of face masks and found that face masks more than doubled the influence of algorithm decisions on human similarity-confidence ratings. We then investigated if this increase in cognitive bias was dependent on perceived algorithm accuracy by also presenting algorithm accuracy rates in the presence of face masks. We found that making humans aware of the potential for algorithm errors mitigated the increase in cognitive bias due to face masks. Our findings suggest that humans reviewing face recognition algorithm decisions should be made aware of the potential for algorithm errors to improve human-algorithm team performance.

Face masks impair facial emotion recognition and induce specific emotion confusions.

Face masks are now worn frequently to reduce the spreading of the SARS-CoV-2 virus. Their health benefits are undisputable, but covering the lower half of one's face also makes it harder for others to recognize facial expressions of emotions. Three experiments were conducted to determine how strongly the recognition of different facial expressions is impaired by masks, and which emotions are confused with each other. In each experiment, participants had to recognize facial expressions of happiness, sadness, anger, surprise, fear, and disgust, as well as a neutral expression, displayed by male and female actors of the Radboud Faces Database. On half of the 168 trials, the lower part of the face was covered by a face mask. In all experiments, facial emotion recognition (FER) was about 20% worse for masked faces than for unmasked ones (68% correct vs. 88%). The impairment was largest for disgust, followed by fear, surprise, sadness, and happiness. It was not significant for anger and the neutral expression. As predicted, participants frequently confused emotions that share activation of the visible muscles in the upper half of the face. In addition, they displayed response biases in these confusions: They frequently misinterpreted disgust as anger, fear as surprise, and sadness as neutral, whereas the opposite confusions were less frequent. We conclude that face masks do indeed cause a marked impairment of FER and that a person perceived as angry, surprised, or neutral may actually be disgusted, fearful, or sad, respectively. This may lead to misunderstandings, confusions, and inadequate reactions by the perceivers.

A Two-Stage Deep Generative Model for Masked Face Synthesis.

Research on face recognition with masked faces has been increasingly important due to the prolonged COVID-19 pandemic. To make face recognition practical and robust, a large amount of face image data should be acquired for training purposes. However, it is difficult to obtain masked face images for each human subject. To cope with this difficulty, this paper proposes a simple yet practical method to synthesize a realistic masked face for an unseen face image. For this, a cascade of two convolutional auto-encoders (CAEs) has been designed. The former CAE generates a pose-alike face wearing a mask pattern, which is expected to fit the input face in terms of pose view. The output of the former CAE is readily fed into the secondary CAE for extracting a segmentation map that localizes the mask region on the face. Using the segmentation map, the mask pattern can be successfully fused with the input face by means of simple image processing techniques. The proposed method relies on face appearance reconstruction without any facial landmark detection or localization techniques. Extensive experiments with the GTAV Face database and Labeled Faces in the Wild (LFW) database show that the two complementary generators could rapidly and accurately produce synthetic faces even for challenging input faces (e.g., low-resolution face of 25 × 25 pixels with out-of-plane rotations).

Emotional-Health-Oriented Urban Design: A Novel Collaborative Deep Learning Framework for Real-Time Landscape Assessment by Integrating Facial Expression Recognition and Pixel-Level Semantic Segmentation.

Emotional responses are significant for understanding public perceptions of urban green space (UGS) and can be used to inform proposals for optimal urban design strategies to enhance public emotional health in the times of COVID-19. However, most empirical studies fail to consider emotion-oriented landscape assessments under dynamic perspectives despite the fact that individually observed sceneries alter with angle. To close this gap, a real-time sentimental-based landscape assessment framework is developed, integrating facial expression recognition with semantic segmentation of changing landscapes. Furthermore, a case study using panoramic videos converted from Google Street View images to simulate changing scenes was used to test the viability of this framework, resulting in five million big data points. The result of this study shows that through the collaboration of deep learning algorithms, finer visual variables were classified, subtle emotional responses were tracked, and better regression results for valence and arousal were obtained. Among all the predictors, the proportion of grass was the most significant predictor for emotional perception. The proposed framework is adaptable and human-centric, and it enables the instantaneous emotional perception of the built environment by the general public as a feedback survey tool to aid urban planners in creating UGS that promote emotional well-being.

Hyperspectral Face Recognition with Adaptive and Parallel SVMs in Partially Hidden Face Scenarios.

Hyperspectral imaging opens up new opportunities for masked face recognition via discrimination of the spectral information obtained by hyperspectral sensors. In this work, we present a novel algorithm to extract facial spectral-features from different regions of interests by performing computer vision techniques over the hyperspectral images, particularly Histogram of Oriented Gradients. We have applied this algorithm over the UWA-HSFD dataset to extract the facial spectral-features and then a set of parallel Support Vector Machines with custom kernels, based on the cosine similarity and Euclidean distance, have been trained on fly to classify unknown subjects/faces according to the distance of the visible facial spectral-features, i.e., the regions that are not concealed by a face mask or scarf. The results draw up an optimal trade-off between recognition accuracy and compression ratio in accordance with the facial regions that are not occluded.

Face masks affect emotion categorisation, age estimation, recognition, and gender classification from faces.

Although putting on a mask over our nose and mouth is a simple but powerful way to protect ourselves and others during a pandemic, face masks may interfere with how we perceive and recognize one another, and hence, may have far-reaching impacts on communication and social interactions. To date, it remains relatively unknown the extent to which wearing a face mask that conceals the bottom part of the face affects the extraction of different facial information. To address this question, we compared young adults' performance between masked and unmasked faces in four different tasks: (1) emotion recognition task, (2) famous face recognition and naming test, (3) age estimation task, and (4) gender classification task. Results revealed that the presence of face mask has a negative impact on famous face recognition and emotion recognition, but to a smaller extent on age estimation and gender classification tasks. More interestingly, we observed a female advantage in the famous face recognition and emotion recognition tasks and a female own-gender bias in gender categorisation and age estimation tasks. Overall, these findings allude to the lack of malleability of the adulthood face recognition and perceptual systems.

Recognition of Masked Faces in the Era of the Pandemic: No Improvement Despite Extensive Natural Exposure.

Face masks, which became prevalent across the globe during the COVID-19 pandemic, have had a negative impact on face recognition despite the availability of critical information from uncovered face parts, especially the eyes. An outstanding question is whether face-mask effects would be attenuated following extended natural exposure. This question also pertains, more generally, to face-recognition training protocols. We used the Cambridge Face Memory Test in a cross-sectional study (N = 1,732 adults) at six different time points over a 20-month period, alongside a 12-month longitudinal study (N = 208). The results of the experiments revealed persistent deficits in recognition of masked faces and no sign of improvement across time points. Additional experiments verified that the amount of individual experience with masked faces was not correlated with the mask effect. These findings provide compelling evidence that the face-processing system does not easily adapt to visual changes in face stimuli, even following prolonged real-life exposure.

MsWH: A Multi-Sensory Hardware Platform for Capturing and Analyzing Physiological Emotional Signals.

This paper presents a new physiological signal acquisition multi-sensory platform for emotion detection: Multi-sensor Wearable Headband (MsWH). The system is capable of recording and analyzing five different physiological signals: skin temperature, blood oxygen saturation, heart rate (and its variation), movement/position of the user (more specifically of his/her head) and electrodermal activity/bioimpedance. The measurement system is complemented by a porthole camera positioned in such a way that the viewing area remains constant. Thus, the user's face will remain centered regardless of its position and movement, increasing the accuracy of facial expression recognition algorithms. This work specifies the technical characteristics of the developed device, paying special attention to both the hardware used (sensors, conditioning, microprocessors, connections) and the software, which is optimized for accurate and massive data acquisition. Although the information can be partially processed inside the device itself, the system is capable of sending information via Wi-Fi, with a very high data transfer rate, in case external processing is required. The most important features of the developed platform have been compared with those of a proven wearable device, namely the Empatica E4 wristband, in those measurements in which this is possible.

A PNU-Based Methodology to Improve the Reliability of Biometric Systems.

Face recognition is an important application of pattern recognition and image analysis in biometric security systems. The COVID-19 outbreak has introduced several issues that can negatively affect the reliability of the facial recognition systems currently available: on the one hand, wearing a face mask/covering has led to growth in failure cases, while on the other, the restrictions on direct contact between people can prevent any biometric data being acquired in controlled environments. To effectively address these issues, we designed a hybrid methodology that improves the reliability of facial recognition systems. A well-known Source Camera Identification (SCI) technique, based on Pixel Non-Uniformity (PNU), was applied to analyze the integrity of the input video stream as well as to detect any tampered/fake frames. To examine the behavior of this methodology in real-life use cases, we implemented a prototype that showed two novel properties compared to the current state-of-the-art of biometric systems: (a) high accuracy even when subjects are wearing a face mask; (b) whenever the input video is produced by deep fake techniques (replacing the face of the main subject) the system can recognize that it has been altered providing more than one alert message. This methodology proved not only to be simultaneously more robust to mask induced occlusions but also even more reliable in preventing forgery attacks on the input video stream.

Emotion Recognition for Partial Faces Using a Feature Vector Technique.

Wearing a facial mask is indispensable in the COVID-19 pandemic; however, it has tremendous effects on the performance of existing facial emotion recognition approaches. In this paper, we propose a feature vector technique comprising three main steps to recognize emotions from facial mask images. First, a synthetic mask is used to cover the facial input image. With only the upper part of the image showing, and including only the eyes, eyebrows, a portion of the bridge of the nose, and the forehead, the boundary and regional representation technique is applied. Second, a feature extraction technique based on our proposed rapid landmark detection method employing the infinity shape is utilized to flexibly extract a set of feature vectors that can effectively indicate the characteristics of the partially occluded masked face. Finally, those features, including the location of the detected landmarks and the Histograms of the Oriented Gradients, are brought into the classification process by adopting CNN and LSTM; the experimental results are then evaluated using images from the CK+ and RAF-DB data sets. As the result, our proposed method outperforms existing cutting-edge approaches and demonstrates better performance, achieving 99.30% and 95.58% accuracy on CK+ and RAF-DB, respectively.

A neural signature of exposure to masked faces after 18 months of COVID-19.

In the last two years, face-to-face interactions have drastically changed worldwide, because of the COVID-19 pandemic: the persistent use of masks has had the advantage of reducing viral transmission, but it has also had the cost of impacting on the perception and recognition of social information from faces, especially emotions. To assess the cerebral counterpart to this condition, we carried out an EEG experiment, extracting Event-Related Potentials (ERPs) evoked by emotional faces with and without surgical masks. Besides the expected impairment in emotion recognition in both accuracy and response times, also the classical face-related ERPs (N170 and P2) are altered by the presence of surgical masks. Importantly, the effect is stronger in individuals with a lower daily exposure to masks, suggesting that the brain must adapt to an extra constraint in decoding social input, due to masks hiding crucial facial information.

New evidence of impaired expression recognition in developmental prosopagnosia.

Developmental prosopagnosia (DP) is a neurodevelopmental condition characterized by lifelong face recognition difficulties. To date, it remains unclear whether or not individuals with DP experience impaired recognition of facial expressions. It has been proposed that DPs may have sufficient perceptual ability to correctly interpret facial expressions when tasks are relatively easy (e.g., the stimuli are unambiguous and viewing conditions are optimal), but exhibit subtle impairments when tested under more challenging conditions. In the present study, we sought to take advantage of the COVID-19 pandemic to test this view. It is well-established that the surgical-type masks worn during the pandemic hinder the recognition and interpretation of facial emotion in typical participants. Relative to typical participants, we hypothesized that DPs may be disproportionately impaired when asked to interpret the facial emotion of people wearing face masks. We compared the ability of 34 DPs and 60 age-matched typical controls to recognize facial emotions i) when the whole face is visible, and ii) when the lower portion of the face is covered with a surgical mask. When expression stimuli were viewed without a mask, the DPs and typical controls exhibited similar levels of performance. However, when expression stimuli were shown with a mask, the DPs showed signs of subtle expression recognition deficits. The DPs were particularly prone to mislabeling masked expressions of happiness as emotion neutral. These results add to a growing body of evidence that under some conditions, DPs do exhibit subtle deficits of expression recognition.

Impact of mask use on face recognition: an eye-tracking study.

We examined how mask use affects performance and eye movements in face recognition and whether strategy change reflected in eye movements is associated with performance change. Eighty-eight participants performed face recognition with masked faces either during learning only, during recognition only, or during both learning and recognition. As compared with the baseline condition where faces were unmasked during both learning and recognition, participants had impaired performance in all three scenarios, with larger impairment when mask conditions during learning and recognition did not match. When recognizing unmasked faces, whether the faces were learned with or without a mask on did not change eye movement behavior. Nevertheless, when recognizing unmasked faces that were learned with a mask on, participants who adopted more eyes-focused patterns had less performance impairment as compared with the baseline condition. When recognizing masked faces, participants had more eyes-focused patterns and more consistent gaze transition behavior than recognizing unmasked faces regardless of whether the faces were learned with or without a mask on. Nevertheless, when recognizing masked faces that were learned without a mask, participants whose gaze transition behavior was more consistent had less performance impairment as compared with the baseline condition. Thus, although eye movements during recognition were mainly driven by the mask condition during recognition but not that during learning, those who adjusted their strategy according to the mask condition difference between learning and recognition had better performance. This finding has important implications for identifying populations vulnerable to the impact of mask use and potential remedial strategies.